Sensor devices, electronic devices, method for performing object detection by a sensor device, and method for performing object detection by an electronic device

ABSTRACT

A sensor device is provided. The sensor device includes an image sensor having a plurality of photo-sensitive pixels configured to measure light received from a scene. The image sensor is configured to output image data indicative of measurement values of at least part of the plurality of photo-sensitive pixels. Additionally, the sensor device includes processing circuitry configured to determine a histogram based on the image data. The histogram represents a distribution of the measurement values. The processing circuitry is further configured to determine whether an object is present in the scene based on the histogram. In addition, the sensor device includes interface circuitry configured to output presence data indicating whether the object is present in the scene.

TECHNICAL FIELD

The present disclosure relates to object detection. In particular,examples relate to sensor devices, electronic devices, a method forperforming object detection by a sensor device and a method forperforming object detection by an electronic device.

BACKGROUND

Power consumption is an important characteristic for imaging devices.For example, it is very important for a mobile phone with face-unlockfunctionality that the mobile phone unlocks immediately when a userfaces the mobile phone. The mobile phone, however, needs to detect thatthe user faces the mobile phone. A significant amount of energy isneeded to determine with conventional imaging devices (e.g. aconventional Time-of-Flight, ToF, sensor) whether the user faces themobile phone.

Hence, there may be a demand for improved object detection.

SUMMARY

The demand may be satisfied by the subject matter of the appendedclaims.

An example relates to a sensor device. The sensor device comprises animage sensor comprising a plurality of photo-sensitive pixels configuredto measure light received from a scene. The image sensor is configuredto output image data indicative of measurement values of at least partof the plurality of photo-sensitive pixels. Additionally, the sensordevice comprises processing circuitry configured to determine ahistogram based on the image data. The histogram represents adistribution of the measurement values. The processing circuitry isfurther configured to determine whether an object is present in thescene based on the histogram. In addition, the sensor device comprisesinterface circuitry configured to output presence data indicatingwhether the object is present in the scene.

Another example relates to an electronic device comprising a sensordevice as described herein and an application processor coupled to thesensor device. The application processor is configured to receive thepresence data and perform an action based on the presence data.

A further example relates to a method for performing object detection bya sensor device. The method comprises measuring light received from ascene by a plurality of photo-sensitive pixels of an image sensor of thesensor device. Further, the method comprises determining, by processingcircuitry of the sensor device, a histogram based on image data outputby the image sensor. The image data are indicative of measurement valuesof at least part of the plurality of photo-sensitive pixels. Thehistogram represents a distribution of the measurement values indicatedby the image data. In addition, the method comprises determining, by theprocessing circuitry, whether an object is present in the scene based onthe histogram. The method further comprises outputting, by interfacecircuitry of the sensor device, presence data indicating whether theobject is present in the scene.

BRIEF DESCRIPTION OF THE FIGURES

Some examples of apparatuses and/or methods will be described in thefollowing by way of example only, and with reference to the accompanyingfigures, in which

FIG. 1 illustrates an example of a sensor device;

FIG. 2 illustrates an exemplary heat map of a first plurality ofhistograms;

FIG. 3 illustrates an exemplary heat map of a second plurality ofhistograms;

FIG. 4 illustrates exemplary errors for two pluralities of histograms;

FIG. 5 illustrates an example of an electronic device;

FIG. 6 illustrates a flowchart of an example of a method for performingobject detection by a sensor device;

FIG. 7 illustrates another example of a sensor device; and

FIG. 8 illustrates a flowchart of an example of a method for performingobject detection by an electronic device.

DETAILED DESCRIPTION

Some examples are now described in more detail with reference to theenclosed figures. However, other possible examples are not limited tothe features of these embodiments described in detail. Other examplesmay include modifications of the features as well as equivalents andalternatives to the features. Furthermore, the terminology used hereinto describe certain examples should not be restrictive of furtherpossible examples.

Throughout the description of the figures same or similar referencenumerals refer to same or similar elements and/or features, which may beidentical or implemented in a modified form while providing the same ora similar function. The thickness of lines, layers and/or areas in thefigures may also be exaggerated for clarification.

When two elements A and B are combined using an “or”, this is to beunderstood as disclosing all possible combinations, i.e. only A, only Bas well as A and B, unless expressly defined otherwise in the individualcase. As an alternative wording for the same combinations, “at least oneof A and B” or “A and/or B” may be used. This applies equivalently tocombinations of more than two elements.

If a singular form, such as “a”, “an” and “the” is used and the use ofonly a single element is not defined as mandatory either explicitly orimplicitly, further examples may also use several elements to implementthe same function. If a function is described below as implemented usingmultiple elements, further examples may implement the same functionusing a single element or a single processing entity. It is furtherunderstood that the terms “include”, “including”, “comprise” and/or“comprising”, when used, describe the presence of the specifiedfeatures, integers, steps, operations, processes, elements, componentsand/or a group thereof, but do not exclude the presence or addition ofone or more other features, integers, steps, operations, processes,elements, components and/or a group thereof.

FIG. 1 illustrates an example of a sensor device 100. The sensor device100 comprises an illumination element (circuitry, device) 130 foremitting (e.g. modulated) light 102 to a scene and an image sensor 110for capturing light 103 received from the scene.

The illumination element 130 generates the (e.g. modulated) light 102.The illumination element 130 may comprise any number of light sources.The illumination element 130 may, e.g., comprise one or moreLight-Emitting Diodes (LEDs) and/or one or more laser diodes (e.g. oneor more Vertical-Cavity Surface-Emitting Lasers, VCSELs) which are firedbased on an illumination signal.

The image sensor may be a two-dimensional (2D) or a three-dimensional(3D) image sensor. The image sensor 110 comprises a plurality ofphoto-sensitive pixels (e.g. comprising a Photonic Mixer Device, PMD, ora Charge-Coupled Device, CCD) configured to measure the light 103received from a scene. The image sensor 110 may comprise variousadditional components such as e.g. optics (e.g. one or more lenses) andelectronic circuitry. The image sensor 110 is configured to output imagedata 111 indicative of measurement values of at least part of theplurality of photo-sensitive pixels.

For various reasons, it may be desirable to know whether a predeterminedobject 101, a predetermined class of objects or a predetermined group(combination) of objects is present in the scene. For example, theobject 101 may be a face of a human being, a predetermined room, a wetroad, a dirty road, rain or fog. However, the present disclosure is notlimited thereto. In general, the object 101 may be any physical object(i.e. anything material that may be perceived by the human senses). Incase the object 101 is present in the scene, the object 101 reflects theemitted light 102 and, hence, generates at least part of the light 103received from the scene.

For enabling detection of the object 101, the sensor device 100comprises processing circuitry 120, which is coupled to the image sensor110. For example, the processing circuitry 120 may be a single dedicatedprocessor, a single shared processor, or a plurality of individualprocessors, some of which or all of which may be shared, a neuromorphicprocessor, a digital signal processor (DSP) hardware, an applicationspecific integrated circuit (ASIC) or a field programmable gate array(FPGA). The processing circuitry 120 may optionally be coupled to, e.g.,read only memory (ROM) for storing software, random access memory (RAM)and/or non-volatile memory. The processing circuitry 120 is configuredto receive the image data 111 and determine a histogram based on theimage data 111.

The histogram represents a distribution of the measurement values. Therange of possible measurement values is divided into a plurality(series) of intervals, which are also known as “bins”, for the histogramsuch that the histogram indicates how many of the measurement valuesfall into each interval (bin). In other words, the number ofphoto-sensitive pixels outputting the same measurement value is countedin each bin of the histogram. The bins are specified as consecutive,non-overlapping intervals of the range of possible measurement values.The bins are adjacent and of equal size. In alternative examples, thebins may be of different size. The granularity of the bins may beselected as desired. For example, a separate bin may be provided foreach possible measurement value in the range of possible measurementvalues. In other examples, a respective single bin may be provided fortwo or more consecutive possible measurement values in the range ofpossible measurement values. In other words, a bin of the histogram mayrepresent a single possible measurement value in the range of possiblemeasurement values or a plurality of consecutive possible measurementvalues in the range of possible measurement values. Any number of binsmay be used (e.g. 40 or 64 bins).

For example, the histogram may be a one-dimensional (1D) vector with aplurality of vector elements (entries). Each vector element (entry)represents a bin, i.e., a respective possible measurement value or arespective plurality of consecutive possible measurement values. Thevalue of the respective vector element (entry) denotes the number ofphoto-sensitive pixels outputting a measurement value identical to thepossible measurement value represented by the respective vector element(entry) or included in the plurality of consecutive possible measurementvalues represented by the respective vector element (entry).

The processing circuitry 120 is further configured to determine whetheran object such as the object 101 is present in the scene based on thehistogram. In particular, the processing circuitry 120 is configured todetermine whether a predetermined object, a predetermined class ofobjects or a predetermined group (combination) of objects is present inthe scene based on the histogram. The object detection based on thehistogram is possible as the histogram exhibits a specific pattern incase a given (predetermined) object, a given (predetermined) class ofobjects or a given (predetermined) group (combination) of objects ispresent in the scene. This is exemplarily illustrated in FIGS. 2 and 3 .

FIG. 2 illustrates a heap map 200 for 160 histograms. Each line of theheat map 200 represents one of the 160 histograms. The abscissa of theheap map 200 denotes the 40 bins of the respective histogram.

In the example of FIG. 2 , a face of a human being was captured underdifferent angles and for different distances by the sensor device 100 soas to generate respective image data. The histograms are generated bythe processing circuitry 120 from the image data. The example of FIG. 2is representative for an environment seen by a mobile device such as amobile phone when a user is operating the mobile device. For example,the example of FIG. 2 may represent the environment seen by a mobiledevice when user wishes to face-unlock the mobile device.

As can be seen from FIG. 2 , all histograms exhibit a similar pattern,i.e., a similar distribution of the measurement values measured by theimage sensor 110. The respective distribution of the measurement valuespeaks between bins 15 to 25 for each histogram. A significantly reducednumber of measurement values falls into the bins 0 to 15 and 25 to 40.

As a comparison, FIG. 3 illustrates another heat map 300 for 60histograms. Each line of the heat map 300 represents one of the 60histograms. The abscissa of the heap map 300 denotes the 40 bins of therespective histogram.

In the example of FIG. 3 , no face was present in the scene contrary tothe example of FIG. 2 . As can be seen from FIG. 3 , the histograms nolonger exhibit a similar pattern. The individual histograms peak atdifferent bin ranges.

A histogram according to the present disclosure does not contain anyinformation about the positions of the image sensor 110'sphoto-sensitive pixels and, hence, does not comprise any geometricalinformation contained in the image data 111. Nevertheless, as can beseen from the examples of FIGS. 2 and 3 , a histogram contains enoughinformation to detect (classify) an object such as a face of a humanbeing. Accordingly, the processing circuitry 120 is able to determinewhether an object such as the object 101 is present in the scene basedon the histogram generated from the image data 111 of the image sensor110.

The processing circuitry 120 outputs presence data 141 indicatingwhether the object 101 is present in the scene. For example, thepresence data 141 may indicate whether a predetermined object or apredetermined class of objects is present in the scene.

In addition, the sensor device 100 comprises interface circuitry 140configured to the output the presence data 141. The presence data 141may be further processes by external circuitry 150 such as anapplication processor of an electronic device comprising the sensordevice 100.

The processing circuitry 120 may determine in various ways whether theobject 101 is present in the scene. For example, the processing circuit120 may be configured to process the histogram by a classificationalgorithm to determine whether the object 101 is present in the scene.In the following two exemplary classification algorithms will bedescribed in detail. However, it is to be noted that the presentdisclosure is not limited thereto and that any other suitableclassification algorithm may be used as well.

In the first exemplary classification algorithm, a reference histogramis used. The reference histogram is generated based on one or morehistogram of the object whose presence in the scene is to be detected.For example, the object may be captured one or more times by the sensordevice 100 such that the image sensor 110 generates reference imagedata. The histograms are generated by the processing circuitry 120 basedon the reference image data output by the image sensor 110. For example,the reference histogram may be formed by the median for each bin of thehistograms generated based on the reference image data. However, it isto be noted that the present disclosure is not limited thereto and thatalso any other suitable combination of the histograms generated based onthe reference image data may be used as well (e.g. weighted averaging ofthe individual bins).

The first exemplary classification algorithm determines as an errormeasure E the sum of the absolute differences between the bins of thehistogram generated from the image data 111 and the bins of thereference histogram:

E=Σ _(i) abs (h _(ref)(i)−h(i))  (1),

with i denoting the number of the respective bin in the histogramgenerated from the image data 111 and the reference histogram, h_(ref)(i) denoting the value of the i-th bin in the reference histogram, h(i)denoting the value of the i-th bin in the histogram generated from theimage data 111, and abs( ) denoting the absolute value function.

Further, the first exemplary classification algorithm determines thatthe object is present in the scene if the error measure E, i.e., the sumof the absolute differences, is below a threshold value T. In case theerror measure E, i.e., the sum of the absolute differences, is above athreshold value T, the first exemplary classification algorithmdetermines that the object is not present in the scene. This may beexpressed as a decision function d:

$\begin{matrix}{{d(E)}:=\left\{ {\begin{matrix}{{1{if}{}E} < T} \\{{0{if}E} > T}\end{matrix},} \right.} & (2)\end{matrix}$

with 1 denoting that the object is present in the scene and 0 denotingthat the object is not present in the scene.

FIG. 4 illustrates an exemplary error plot. In the example of FIG. 4 ,the curve 410 represents the respective error measure E, i.e., the sumof the absolute differences for 130 histograms of a first data set.Similarly, the curve 420 represents the respective error measure E,i.e., the sum of the absolute differences for 55 histograms of a seconddata set. The abscissa denotes the number of the respective histogram.The ordinate denotes the error measure E for the respective histogram.The line 430 represents the threshold value T.

A face of a human being was captured under different angles and fordifferent distances by the sensor device 100 for the histograms in thefirst data set. No face was present in the scene for histograms in thesecond data.

As can be seen from FIG. 4 , the respective error measure E is below thethreshold value T for all histograms of the first data set. Accordingly,it is determined for all histograms of the first data set that the faceis present in the scene. None of the histograms of the first data set isclassified falsely.

Also, almost all histograms of the second data set are classifiedcorrectly. The respective error measure E is above the threshold value Tfor all but four histograms of the second data set. Accordingly, it isdetermined for all but four histograms of the second data set that theface is not present in the scene. The false classification of only fourhistograms of the second data set is acceptable.

The threshold value T may be set manually. Alternatively, the thresholdvalue T may be determined from the reference image data output by theimage sensor 110. For example, the threshold value T may be learned whengenerating the reference histogram. The training of the first exemplaryclassification algorithm may be performed by the processing circuitry120. In other words, processing circuitry 120 may be configured to trainthe classification algorithm based on the reference image data output bythe image sensor 110.

Also in the second exemplary classification algorithm, a referencehistogram is used. The reference histogram may be generated as describedabove. The second exemplary classification algorithm determines anintersection S of the reference histogram and the histogram generatedfrom the image data 111:

S=Σ _(i) min(h _(ref)(i),h(i))  (3),

with min( ) denoting the minimum function returning the smallest of itsarguments (inputs).

The more similar the reference histogram and the histogram generatedfrom the image data 111 are, the greater is the intersection S. Hence,the intersection S may, similar to what is described above for the firstexemplary classification algorithm, be compared to threshold in order todetermine whether the object is present in the scene. Optionally, theintersection S may be normalized to a value range between 0 and 1:

$\begin{matrix}{{S_{normal} = \frac{\sum\limits_{i}{\min\left( {{h_{ref}(i)},{h(i)}} \right)}}{\sum\limits_{i}{h_{ref}(i)}}},} & (4)\end{matrix}$

with S_(normal) denoting the normalized intersection. The normalizedintersection S_(normal) may again be compared to a threshold value. Ifthe normalized intersection S_(normal) is below the threshold value, thesecond exemplary classification algorithm determines that the object isnot present in the scene. In case the normalized intersection S_(normal)is above the threshold value, the second exemplary classificationalgorithm determines that the object is present in the scene.

Both exemplary classification algorithms determine whether apredetermined object such as the object 101, a predetermined class ofobjects or a predetermined group (combination) of objects is present inthe scene based on a comparison of the histogram generated from theimage data 111 to a reference histogram.

The classification of the histogram generated from the image data 111requires only little computing power and, may, hence be done on-chip. Inother words, the sensor device 100 may comprises a semiconductor diecomprising both the image sensor 110 and the processing circuitry 120.

As described above, the image sensor 110 may, in general, be any kind of2D or 3D image sensor. The respective measurement value ofphoto-sensitive pixel of the image sensor 110 indicates a respectiveamount of light received by the photo-sensitive pixel (for example, theimage data 111 may represent an 2D grayscale image of at least part ofthe scene). As described above, the image data 111 indicate themeasurement values of at least part of the plurality of photo-sensitivepixels of the image sensor 110. It is to be noted that the image dataindicate the measurement values of the at least part of the plurality ofphoto-sensitive pixels for a single exposure of the plurality ofphoto-sensitive pixels.

In some examples, at least one of the plurality of photo-sensitivepixels of the image sensor 110 may be configured to selectively storecharge carriers generated by the light received from the scene insemiconductor material of the at least one of the plurality ofphoto-sensitive pixels in different charge storages or a drain node ofthe at least one of the plurality of photo-sensitive pixels over time.Accordingly, the respective measurement value of the at least one of theplurality of photo-sensitive pixels is based on the charge carriersstored in at least one of the different charge storages or the drainnode. In other examples, at least one of the plurality ofphoto-sensitive pixels of the image sensor 110 may be configured togenerate its respective measurement value based on a correlation of thelight 103 received from the scene with a reference signal used fordriving the at least one of the plurality of photo-sensitive pixels. Theimage sensor may, e.g., be a ToF sensor operating according to one ofthe above principles.

In case the image sensor 110 is a ToF sensor, at least one of theplurality of photo-sensitive pixels may be configured to measure thelight 103 received from the scene using a (light-intensity-independent)correlation function that increases (e.g. strictly monotonic) overdistance within a target measurement range of the ToF sensor 110. Inother words, parameters of the at least one of the plurality ofphoto-sensitive pixels may be adjusted such that the(light-intensity-independent) correlation function increases (e.g.strictly monotonic) over distance within the target measurement range ofthe ToF sensor 110. The (light-intensity-independent) correlationfunction gives the photo-sensitive pixel's distance-dependentcorrelation of the received light 103 with the reference signal andwithout considering (i.e. ignoring, not taking into account) theintensity of the received light 103. As described above, thephoto-sensitive pixel is being driven based on the reference signal. Inother words, the (light-intensity-independent) correlation function onlydescribes the distance-dependency of the photo-sensitive pixel's output(i.e. the dependency of the photo-sensitive pixel's output on thedistance between the ToF sensor 110 and the object 101) but not thedependency of the photo-sensitive pixel's output on the intensity of thereceived light 103. The respective (light-intensity-independent)correlation function of the other photo-sensitive pixels of the ToFsensor 110 may be adjusted as described above.

The intensity (light strength) of the light 103 received from the object101 in the scene is decreasing over the distance between the ToF sensor110 and the object 101. For example, it may be assumed that theintensity decreases according to the inverse square law. That is, thedistance-dependent intensity of the light 103 received at the ToF sensor110 may be assumed as follows:

$\begin{matrix}{{I(d)} \propto \frac{1}{d^{2}}} & (5)\end{matrix}$

with I denoting the intensity of the light 103 received at the ToFsensor 110 and d denoting distance between the ToF sensor 110 and theobject 101 reflecting the emitted light 102 back to the ToF sensor 110.

Accordingly, the (light-intensity-independent) correlation function c(d)may, e.g., be adjusted to increase with the square of the distance d:

c(d)∝d ²  (6)

The square increase of the (light-intensity-independent) correlationfunction is a good approximation in case the object 101 behave likes apoint-like light source.

The (actual) measurement value of the at least one of the plurality ofphoto-sensitive pixels of the ToF sensor 110 scales with the intensityof the light 103 received at the photo-sensitive pixel (i.e. the lightstrength of the light 103 from the object 101). For example, themeasurement value of the at least one of the plurality ofphoto-sensitive pixels of the ToF sensor 110 may be determined by theproduct of the intensity of the light 103 received at thephoto-sensitive pixel and the value of the (light-intensity-independent)correlation function at the distance of the object 101 reflecting thelight 103 to the photo-sensitive pixel.

As the (light-intensity-independent) correlation function increases(e.g. strictly monotonic) over distance, the decreasing light intensityover distance may be counteracted. Accordingly, a brightness of theobject in the image data 111 does not depend on the distance between theobject 101 and the ToF sensor 110. The measurement values of the ToFsensor 110's photo-sensitive pixels are proportional to the reflectivityof the object 101 as the reflectivity of the object 101 determines howmuch light arrives at the ToF sensor 110. Accordingly, the measurementvalues of the ToF sensor 110's photo-sensitive pixels vary with thereflectivity of the object 101—independent of the distance between theToF sensor 110 and the object 101. Therefore, the measurement valuesindicated by the image data 110 represent the reflectivity of the object101.

Many different modulation patterns for the emitted light 102 as well asthe reference signal for driving the ToF sensor 110 may be used toobtain the above-described shape of the (light-intensity-independent)correlation function.

The above described image data 111 may be analog or digitaldata—independent of the specific implementation of the image sensor 110.For example, read-out circuitry of the image sensor 110 may beconfigured to read out at least part of the plurality of photo-sensitivepixels to obtain the measurement values. In some examples, all of theplurality of photo-sensitive pixels may be read out by the read-outcircuitry. In other examples, only part (i.e. only a subset) of theplurality of photo-sensitive pixels may be read out by the read-outcircuitry. For example, the photo-sensitive pixels may be read out bythe read-out circuitry according to one or more (e.g. predefined)patterns. For example, pixel read-out may be skipped according to apattern. Omitting the read-out of some of the photo-sensitive pixels mayallow to reduce the energy consumption of the sensor device 100. Theread-out measurement values may be analog values.

The analog measurement values may be digitized by means of anAnalog-to-Digital Converter (ADC) of the image sensor 110. Accordingly,the processing circuitry 120 may be configured to determine thehistogram based on the digitized measurement values. For example, theprocessing circuitry may use a set of counters for the bins of thehistogram and increment the respective counter each time a certainmeasurement (pixel) value is digitized. Similar to what is described, arespective counter may be increased if a digitized measurement value iswithin a certain value range. As described above for the bins of thehistogram, the value ranges may be evenly distributed among thedigitized output range of the measurement values, or be optimized withindividual ranges.

The ADC may support (provide) different resolutions for digitizinganalog data. For example, the ADC may support at least a firstresolution and a second resolution for digitizing analog data. The firstresolution is lower than the second resolution. The ADC may beconfigured to digitize the measurement values using the firstresolution. The first resolution may, e.g., match the number ofhistogram bins (e.g. the first resolution may be 6 bit for 2⁶=64 bins).The second resolution may be used by the ADC for digitizing furthermeasurement values of the image sensor 110 that are obtained by theimage sensor 110 for one or more further measurements of the scene (e.g.a ToF depth/distance measurement). Using the lower resolution of the ADCmay be sufficient for the histogramming.

The sensor device 100 may comprise further hardware—conventional and/orcustom. The elements of the sensor device 100 are (all) arranged(packaged) within a housing of the sensor device 100.

The sensor device 100 may, e.g., be used in an electronic device. Anexemplary electronic device 500 (e.g. mobile phone, smartphone,tablet-computer, or laptop-computer) comprising a sensor device 510 asdescribed above is illustrated in FIG. 5 .

The electronic device 500 further comprises an application processor 520coupled to the interface circuitry of the sensor device 510. Theapplication processor 520 is configured to receive the presence dataoutput by the sensor device 510 and to perform an action based on thepresence data. For example, the application processor 520 may beconfigured to launch a wake up procedure for waking up from a sleep modeand/or a lower power mode. Alternatively or additionally, theapplication processor 520 may launch an application or turn on/off adisplay 530 of the electronic device 500.

For example, if interface circuitry of the sensor device 510 outputspresence data 141 indicating that a face of a human being is present inthe scene, the application processor may wake up and launch a faceunlock application for unlocking the electronic device 500. The faceunlock application may require reflectivity data of the object presentin the scene (e.g. a grayscale image) for the face recognition.Accordingly, the application processor 520 may transmit trigger data tothe interface circuitry of the sensor device 510.

In response to receiving the trigger data the image sensor of the sensordevice 510 may capture the scene to generate further image data. This isfurther illustrated in detail in FIG. 1 , in which the trigger data 151are received from the external circuitry 150 such as the applicationprocessor 520 by the interface circuitry 140 and used to control theimage sensor 110 and the illumination element 130. The further imagedata 112 generated by the image sensor 110 are output by the interfacecircuitry 140. Accordingly, the external circuitry external circuitry150 such as the application processor 520 may process the further imagedata by means of, e.g., the face unlock application.

However, it is to be noted that the present disclosure is not limited tothe above exemplary actions performed by the application processor 520.In general, the application processor 520 may perform any action basedon the presence data output by the sensor device 510. For example, theapplication processor 520 may be configured to turn off the display 530in case the presence data output by the sensor device 510 indicate thatno face of a human being is present in the scene.

The sensor device 510 according to the present disclosure is anenergy-efficient solution to detect with simple (e.g. on-chip)processing whether a certain type of object (e.g. a human face) ispresent in an image of a scene. As described above, this information maybe used by the electronic device 500 (e.g. a smartphone) to triggervarious actions such as a face-unlock process including face-recognition(i.e. depth) measurements. Due to the sensor device 510, the applicationprocessor 520 does not need to perform any action for the objectiondetermination.

Although not illustrated in FIG. 5 , the electronic device 500 mayoptionally comprise further circuitry/elements such as, e.g., one ormore microphones, one or more loudspeakers, one or more antennas, one ormore radio frequency transmitters and/or receivers for mobilecommunication, one or more data storages, one or more batteries, etc.

A flowchart of an example of a method 600 for performing objectdetection by a sensor device according to the proposed technique isfurther illustrated in FIG. 6 . The method 600 comprises measuring 602light received from a scene by a plurality of photo-sensitive pixels ofan image sensor of the sensor device. Further, the method 600 comprisesdetermining 604, by processing circuitry of the sensor device, ahistogram based on image data output by the image sensor. The image dataare indicative of measurement values of at least part of the pluralityof photo-sensitive pixels. The histogram represents a distribution ofthe measurement values indicated by the image data. In addition, themethod 600 comprises determining 606, by the processing circuitry,whether an object, a predetermined class of objects or a predeterminedgroup (combination) of objects is present in the scene based on thehistogram. The method 600 further comprises outputting 608, by interfacecircuitry of the sensor device, presence data indicating whether theobject is present in the scene.

The method 600 may allow (e.g. on-chip) object detection with low powerconsumption. The direct creation of the histogram from the image sensorreadout may allow to significantly reduce the image data and enable fastand simple object detection.

More details and aspects of the method 600 are explained in connectionwith the proposed technique or one or more examples described above. Themethod 600 may comprise one or more additional optional featurescorresponding to one or more aspects of the proposed technique or one ormore examples described above.

In the above examples, the object detection is done within the sensordevice. However, the present disclosure is not limited thereto. In someexamples, the object detection may be performed by external circuitry.This is exemplarily illustrated in FIG. 7 . FIG. 7 illustrates anothersensor device 700. The sensor device 700 differs from the abovedescribed sensor device 100 in that the processing circuitry 720 isconfigured to only determine the histogram based on the image data. Incontrast to the processing circuitry 120 of the sensor device 100, theprocessing circuitry 720 does not determine whether a predeterminedobject, a predetermined class of objects or a predetermined group(combination) of objects is present in the scene based on the histogram.The interface circuitry 740 is accordingly configured to outputhistogram data 741 indicative of the histogram. The histogram data 741is output to external circuitry 750 such as, e.g., the applicationprocessor 520 of the electronic device 500 illustrated in FIG. 5 . Theimage data 111 are not output to the external circuitry 750. Other thanthat, the sensor device 700 is identical to the above described sensordevice 100 and may, hence, exhibit one or more of the above describedfeatures of the sensor device 100.

In the example of FIG. 7 , the determination whether the predeterminedobject, the predetermined class of objects or the predetermined group(combination) of objects is present in the scene based on the histogramis performed by the external circuitry 750 such as, e.g., theapplication processor 520 of the electronic device 500 illustrated inFIG. 5 . Accordingly, the sensor device 510 illustrated in FIG. 5 may bethe sensor device 700 in some examples. In other words, the applicationprocessor 520 of the electronic device 500 illustrated in FIG. 5 isconfigured to receive the histogram data from the sensor device 510 andto determine whether the predetermined object, the predetermined classof objects or the predetermined group (combination) of objects ispresent in the scene based on the histogram according to some examples.The determination whether the predetermined object, the predeterminedclass of objects or the predetermined group (combination) of objects ispresent in the scene is done according to the above describedprinciples. Other than that, the application processor 520 is configuredas described above (e.g. to send the above described trigger data to thesensor device 510 if the object is present in the scene).

In order to summarize the proposed distributed object detection, FIG. 8illustrates a flowchart of an example of a method 800 for performingobject detection by an electronic device. The electronic devicecomprises a sensor device and an application processor. The method 800comprises measuring 802 light received from a scene by a plurality ofphoto-sensitive pixels of an image sensor of the sensor device. Further,the method 800 comprises determining 804, by processing circuitry of thesensor device, a histogram based on image data output by the imagesensor. The image data are indicative of measurement values of at leastpart of the plurality of photo-sensitive pixels. The histogramrepresents a distribution of the measurement values indicated by theimage data. In addition, the method 800 comprises outputting 806, byinterface circuitry of the sensor device, histogram data indicative ofthe histogram. The method 800 further comprises receiving 808 thehistogram data at the application processor and determining 810, by theapplication processor, whether a predetermined object, a predeterminedclass of objects or a predetermined group (combination) of objects ispresent in the scene based on the histogram.

The method 800 may allow distributed object detection with low powerconsumption. The direct creation of the histogram from the image sensorreadout may allow to significantly reduce the image data and enable fastand simple object detection.

More details and aspects of the method 800 are explained in connectionwith the proposed technique or one or more examples described above. Themethod 800 may comprise one or more additional optional featurescorresponding to one or more aspects of the proposed technique or one ormore examples described above.

The examples as described herein may be summarized as follows:

Examples relate to a sensor device. The sensor device comprises an imagesensor comprising a plurality of photo-sensitive pixels configured tomeasure light received from a scene. The image sensor is configured tooutput image data indicative of measurement values of at least part ofthe plurality of photo-sensitive pixels. Additionally, the sensor devicecomprises processing circuitry configured to determine a histogram basedon the image data. The histogram represents a distribution of themeasurement values. The processing circuitry is further configured todetermine whether an object is present in the scene based on thehistogram. In addition, the sensor device comprises interface circuitryconfigured to output presence data indicating whether the object ispresent in the scene.

In some examples, the sensor device further comprises a semiconductordie comprising the image sensor and the processing circuitry.

According to some examples, the image sensor comprises: read-outcircuitry configured to read out the at least part of the plurality ofphoto-sensitive pixels to obtain the measurement values; and an ADCconfigured to digitize the measurement values, wherein the processingcircuitry is configured to determine the histogram based on thedigitized measurement values.

In some examples, the ADC supports at least a first resolution and asecond resolution for digitizing analog data, the first resolution beinglower than the second resolution, and wherein the ADC is configured todigitize the measurement values using the first resolution.

According to some examples, at least one of the plurality ofphoto-sensitive pixels is configured to selectively store chargecarriers generated by the light received from the scene in semiconductormaterial of the at least one of the plurality of photo-sensitive pixelsin different charge storages of the at least one of the plurality ofphoto-sensitive pixels over time, and wherein the respective measurementvalue of the at least one of the plurality of photo-sensitive pixels isbased on the charge carriers stored in at least one of the differentcharge storages.

In some examples, the image sensor is a ToF sensor.

According to some examples, at least one of the plurality ofphoto-sensitive pixels is configured to measure the light received fromthe scene using a correlation function that increases over distance.

In some examples, the correlation function gives the photo-sensitivepixel's distance-dependent correlation of the light with a referencesignal without considering the intensity of the light, thephoto-sensitive pixel being driven based on the reference signal.

According to some examples, the processing circuitry is configured toprocess the histogram by a classification algorithm to determine whetherthe object is present in the scene.

In some examples, processing circuitry is further configured to trainthe classification algorithm based on reference image data output by theimage sensor.

According to some examples, the image data indicate the measurementvalues of the at least part of the plurality of photo-sensitive pixelsfor a single exposure of the plurality of photo-sensitive pixels.

In some examples, the measurement values indicate a respective amount oflight received by the at least part of the plurality of photo-sensitivepixels.

According to some examples, the object is a face of a human being.

In some examples, in response to outputting presence data indicatingthat the object is present in the scene, the interface circuitry isconfigured to receive trigger data from external circuitry, wherein theimage sensor is configured to capture the scene to generate furtherimage data in response to receiving the trigger data, and wherein theinterface circuitry is configured to output the further image data.

Other examples relate to an electronic device comprising a sensor deviceas described herein and an application processor coupled to the sensordevice. The application processor is configured to receive the presencedata and perform an action based on the presence data

According to some examples, the electronic device is one of a mobilephone, a tablet-computer or a laptop-computer.

Further examples relate to a method for performing object detection by asensor device. The method comprises measuring light received from ascene by a plurality of photo-sensitive pixels of an image sensor of thesensor device. Further, the method comprises determining, by processingcircuitry of the sensor device, a histogram based on image data outputby the image sensor. The image data are indicative of measurement valuesof at least part of the plurality of photo-sensitive pixels. Thehistogram represents a distribution of the measurement values indicatedby the image data. In addition, the method comprises determining, by theprocessing circuitry, whether an object is present in the scene based onthe histogram. The method further comprises outputting, by interfacecircuitry of the sensor device, presence data indicating whether theobject is present in the scene.

Still other examples relate to another sensor device. The other sensordevice comprises an image sensor comprising a plurality ofphoto-sensitive pixels configured to measure light received from ascene. The image sensor is configured to output image data indicative ofmeasurement values of at least part of the plurality of photo-sensitivepixels. Additionally, the other sensor device comprises processingcircuitry configured to determine a histogram based on the image data.The histogram represents a distribution of the measurement valuesindicated by the image data. Further, the other sensor device comprisesinterface circuitry configured to output histogram data indicative ofthe histogram.

In some examples, the other sensor device further comprises asemiconductor die comprising the image sensor and the processingcircuitry.

According to some examples, the image sensor comprises: read-outcircuitry configured to read out the at least part of the plurality ofphoto-sensitive pixels to obtain the measurement values; and an ADCconfigured to digitize the measurement values, wherein the processingcircuitry is configured to determine the histogram based on thedigitized measurement values.

In some examples, the ADC supports at least a first resolution and asecond resolution for digitizing analog data, the first resolution beinglower than the second resolution, and wherein the ADC is configured todigitize the measurement values using the first resolution.

According to some examples, at least one of the plurality ofphoto-sensitive pixels is configured to selectively store chargecarriers generated by the light received from the scene in semiconductormaterial of the at least one of the plurality of photo-sensitive pixelsin different charge storages of the at least one of the plurality ofphoto-sensitive pixels over time, and wherein the respective measurementvalue of the at least one of the plurality of photo-sensitive pixels isbased on the charge carriers stored in at least one of the differentcharge storages.

In some examples, the image sensor is a ToF sensor.

According to some examples, at least one of the plurality ofphoto-sensitive pixels is configured to measure the light received fromthe scene using a correlation function that increases over distance.

In some examples, the correlation function gives the photo-sensitivepixel's distance-dependent correlation of the light with a referencesignal without considering the intensity of the light, thephoto-sensitive pixel being driven based on the reference signal.

According to some examples, the image data indicate the measurementvalues of the at least part of the plurality of photo-sensitive pixelsfor a single exposure of the plurality of photo-sensitive pixels.

In some examples, the measurement values indicate a respective amount oflight received by the at least part of the plurality of photo-sensitivepixels.

According to some examples, the interface circuitry is furtherconfigured to receive trigger data from external circuitry, wherein theimage sensor is configured to capture the scene to generate furtherimage data in response to receiving the trigger data, and wherein theinterface circuitry is configured to output the further image data.

Still further examples relate to another electronic device comprisingthe other sensor device as described herein and an application processorcoupled to the other sensor device. The application processor isconfigured to receive the histogram data and determine whether an objectis present in the scene based on the histogram.

In some examples, the application processor is further configured toperform an action based on whether the object is present in the scene.

According to some examples, the application processor is furtherconfigured to send trigger data to the sensor device if the object ispresent in the scene, wherein the image sensor is configured to capturethe scene to generate further image data in response to receiving thetrigger data, wherein the interface circuitry is configured to outputthe further image data, and wherein the application processor isconfigured to perform an action based on the further image data.

In some examples, the application processor is configured to process thehistogram by a classification algorithm to determine whether the objectis present in the scene.

According to some examples, the application processor is furtherconfigured to train the classification algorithm based on referenceimage data received from the sensor device.

In some examples, the object is a face of a human being.

According to some examples, the electronic device is one of a mobilephone, a tablet-computer or a laptop-computer.

Examples further relate to another method for performing objectdetection by an electronic device, wherein the electronic devicecomprises a sensor device and an application processor. The methodcomprises measuring light received from a scene by a plurality ofphoto-sensitive pixels of an image sensor of the sensor device. Further,the method comprises determining, by processing circuitry of the sensordevice, a histogram based on image data output by the image sensor. Theimage data are indicative of measurement values of at least part of theplurality of photo-sensitive pixels. The histogram represents adistribution of the measurement values indicated by the image data. Inaddition, the method comprises outputting, by interface circuitry of thesensor device, histogram data indicative of the histogram. The methodfurther comprises receiving the histogram data at the applicationprocessor and determining, by the application processor, whether anobject is present in the scene based on the histogram.

Example of the present disclosure may provide a histogram based objectdetection mode for an image sensor. Some of the examples may provide anenergy efficient on-chip method for mobile phones ToF sensors to detectfaces.

The aspects and features described in relation to a particular one ofthe previous examples may also be combined with one or more of thefurther examples to replace an identical or similar feature of thatfurther example or to additionally introduce the features into thefurther example.

It is further understood that the disclosure of several steps,processes, operations or functions disclosed in the description orclaims shall not be construed to imply that these operations arenecessarily dependent on the order described, unless explicitly statedin the individual case or necessary for technical reasons. Therefore,the previous description does not limit the execution of several stepsor functions to a certain order. Furthermore, in further examples, asingle step, function, process or operation may include and/or be brokenup into several sub-steps, -functions, -processes or -operations.

If some aspects have been described in relation to a device or system,these aspects should also be understood as a description of thecorresponding method. For example, a block, device or functional aspectof the device or system may correspond to a feature, such as a methodstep, of the corresponding method. Accordingly, aspects described inrelation to a method shall also be understood as a description of acorresponding block, a corresponding element, a property or a functionalfeature of a corresponding device or a corresponding system.

The following claims are hereby incorporated in the detaileddescription, wherein each claim may stand on its own as a separateexample. It should also be noted that although in the claims a dependentclaim refers to a particular combination with one or more other claims,other examples may also include a combination of the dependent claimwith the subject matter of any other dependent or independent claim.Such combinations are hereby explicitly proposed, unless it is stated inthe individual case that a particular combination is not intended.Furthermore, features of a claim should also be included for any otherindependent claim, even if that claim is not directly defined asdependent on that other independent claim.

What is claimed is:
 1. A sensor device, comprising: an image sensorcomprising a plurality of photo-sensitive pixels configured to measurelight received from a scene, wherein the image sensor is configured tooutput image data indicative of measurement values of at least part ofthe plurality of photo-sensitive pixels; and processing circuitryconfigured to determine a histogram based on the image data, wherein thehistogram represents a distribution of the measurement values, anddetermine whether an object is present in the scene based on thehistogram; and interface circuitry configured to output presence dataindicating whether the object is present in the scene.
 2. The sensordevice of claim 1, further comprising a semiconductor die comprising theimage sensor and the processing circuitry.
 3. The sensor device of claim1, wherein the image sensor comprises: read-out circuitry configured toread out the at least part of the plurality of photo-sensitive pixels toobtain the measurement values; and an analog-to-digital converter (ADC)configured to digitize the measurement values, wherein the processingcircuitry is configured to determine the histogram based on thedigitized measurement values.
 4. The sensor device of claim 3, whereinthe ADC supports at least a first resolution and a second resolution fordigitizing analog data, the first resolution being lower than the secondresolution, and wherein the ADC is configured to digitize themeasurement values using the first resolution.
 5. The sensor device ofclaim 1, wherein at least one of the plurality of photo-sensitive pixelsis configured to selectively store charge carriers generated by thelight received from the scene in semiconductor material of the at leastone of the plurality of photo-sensitive pixels in different chargestorages of the at least one of the plurality of photo-sensitive pixelsover time, and wherein the respective measurement value of the at leastone of the plurality of photo-sensitive pixels is based on the chargecarriers stored in at least one of the different charge storages.
 6. Thesensor device of claim 1, wherein the image sensor is a time-of-flight(ToF) sensor.
 7. The sensor device of claim 6, wherein at least one ofthe plurality of photo-sensitive pixels is configured to measure thelight received from the scene using a correlation function thatincreases over distance.
 8. The sensor device of claim 7, wherein thecorrelation function gives the photo-sensitive pixel'sdistance-dependent correlation of the light with a reference signalwithout considering the intensity of the light, the photo-sensitivepixel being driven based on the reference signal.
 9. The sensor deviceof claim 1, wherein the processing circuitry is configured to processthe histogram by a classification algorithm to determine whether theobject is present in the scene.
 10. The sensor device of claim 9,wherein the processing circuitry is configured to train theclassification algorithm based on reference image data output by theimage sensor.
 11. The sensor device of claim 1, wherein the image dataindicate the measurement values of the at least part of the plurality ofphoto-sensitive pixels for a single exposure of the plurality ofphoto-sensitive pixels.
 12. The sensor device of claim 1, wherein themeasurement values indicate a respective amount of light received by theat least part of the plurality of photo-sensitive pixels.
 13. The sensordevice of claim 1, wherein in response to outputting presence dataindicating that the object is present in the scene, the interfacecircuitry is configured to receive trigger data from external circuitry,wherein the image sensor is configured to capture the scene to generatefurther image data in response to receiving the trigger data, andwherein the interface circuitry is configured to output the furtherimage data.
 14. An electronic device, comprising: the sensor device ofclaims 1; and an application processor coupled to the sensor device,wherein the application processor is configured to receive the presencedata and perform an action based on the presence data.
 15. A method forperforming object detection by a sensor device, the method comprising:measuring light received from a scene by a plurality of photo-sensitivepixels of an image sensor of the sensor device; determining, byprocessing circuitry of the sensor device, a histogram based on imagedata output by the image sensor, wherein the image data are indicativeof measurement values of at least part of the plurality ofphoto-sensitive pixels, and wherein the histogram represents adistribution of the measurement values indicated by the image data;determining, by the processing circuitry, whether an object is presentin the scene based on the histogram; and outputting, by interfacecircuitry of the sensor device, presence data indicating whether theobject is present in the scene.